Papers with VQA datasets
SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM (2024.findings-emnlp)
Copied to clipboard
Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul Crook, Yifan Xu, Babak Damavandi, Xin Dong, Christos Faloutsos, Lei Li, Seungwhan Moon
| Challenge: | Vision-extended LLMs have made significant strides in VQA, but they still encounter significant difficulties in handling queries involving long-tail entities. |
| Approach: | They propose a benchmark to test models' ability to identify entities and provide detailed, entity-specific knowledge by combining 10 images and 10 knowledge-intensive QA pairs. |
| Outcome: | The proposed model outperforms existing methods on the SnapNTell dataset, achieving a 66.5% improvement in the BELURT score. |
ConceptBert: Concept-Aware Representation for Visual Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. |
| Approach: | They propose an algorithm which learns a joint Concept-Vision-Language embedding for questions which require common sense knowledge from external structured content. |
| Outcome: | The proposed model is based on the Outer Knowledge-VQA and VQA datasets. |
NegVQA: Can Vision Language Models Understand Negation? (2025.findings-acl)
Copied to clipboard
| Challenge: | NegVQA is a visual question answering (VQA) benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Approach: | They propose a visual question answering benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. |
| Outcome: | The proposed model fails to correctly interpret negation, leading to critical errors in interactive AI systems. |
VQAGuider: Guiding Multimodal Large Language Models to Answer Complex Video Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) can grasp the intention of a question and decomposing it to a series of visual recognition sub-tasks to find out the answer with the help of an agent. |
| Approach: | They propose a framework for multimodal large language models to grasp the intention of a question and decompose it into a series of visual recognition sub-tasks to find out the answer. |
| Outcome: | The proposed framework improves the accuracy of complex video-related questions by 29.6% and 17.2% on CVQA and the existing VQA datasets. |
MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering (2025.findings-acl)
Copied to clipboard
Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, An-Lan Wang, Chunhui Lin, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, Can Huang
| Challenge: | Text-Centric Visual Question Answering (TEC-VQA) is a text-centric visual task understanding tool. |
| Approach: | They introduce a benchmark that features human expert annotations across 9 languages . they prioritize the text in question-answer pairs while disregarding visual text in images . |
| Outcome: | The proposed benchmarks prioritize the text in question-answer pairs while disregarding visual text in images. |
X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQA (2025.emnlp-main)
Copied to clipboard
| Challenge: | Medical visual question answering (VQA) and federated learning (FL) are important tools for privacy-preserving collaborative learning. |
| Approach: | They propose a cross-modal FL framework that uses modality-expert low-rank adaptation for medical visual question answering (VQA) X-FLoRA enables the synthesis of images from one modality to another without requiring data sharing . |
| Outcome: | Experiments show that X-FLoRA outperforms existing FL methods in terms of performance . XFLorage enables synthesis of images from one modality to another without data sharing . |
Learning to Ask Denotative and Connotative Questions for Knowledge-based VQA (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have attracted increasing attention due to their prominent performance on various tasks. |
| Approach: | They propose to let LLMs learn to ask informative questions to collect visual information . they introduce concepts of denotation and connotation to promote image and question understanding . |
| Outcome: | The proposed model can generate high-quality questions and efficiently collect required information without expensive training or annotations. |
NLKI: A Lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Small vision-language models lag behind their larger generative counterparts due to lack of knowledge. |
| Approach: | They propose a framework that integrates commonsense knowledge into small vision-language models . the framework retrieves natural language facts and prompts an LLM to craft natural language explanations . |
| Outcome: | The proposed framework retrieves natural language facts and prompts an LLM to craft natural language explanations. |
Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multi-modal language models with different architectures, parameter sizes, training datasets, and pipelines exhibit varying strengths across different tasks. |
| Approach: | They propose a framework for fusing heterogeneous models off-the-shell, which they call likelihood composition, and introduce basic operations to compose multiple models’ likelihood distribution when doing a multi-choice visual-question-answering task. |
| Outcome: | The proposed framework can be used to fusing heterogeneous models off-the-shell. |
Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation (D19-1)
Copied to clipboard
| Challenge: | interacting with a model for Visual Question Answering (VQA) quickly reveals that these models lack consistency. |
| Approach: | They propose a dataset, ConVQA, and metrics that enable quantitative evaluation of consistency in VQA. |
| Outcome: | The proposed data augmentation module improves the consistency of VQA models on the Con-VQA dataset and is a strong baseline for future research. |
TVQACML: Benchmarking Text-Centric Visual Question Answering in Multilingual Chinese Minority Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TEC-VQA benchmarks focus on high-resource languages like English and Chinese . existing benchmarks have a "visual-textual misalignment" problem resulting in unreliable evaluation results . |
| Approach: | They propose a benchmark that expands multilingual QA pairs in non-text-centric datasets through translation to eight languages, including Standard Chinese, Korean, and six minority languages. |
| Outcome: | The proposed benchmarks are contamination-free and more challenging . they include eight languages including Chinese, Korean, and six minority languages . |
Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. |
| Approach: | They propose to use visual-question-answering (VQA) datasets to annotate a 5k-sample VQA preference dataset and to investigate the degradation of VQA datasets. |
| Outcome: | The proposed model surpasses the instruction-following capabilities of the language model with DPO and SteerLM. |
Towards One-to-Many Visual Question Answering (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Visual Question Answering systems are constrained to support domain-specific questions . a model trained on a single specific domain may not be competent for real-world application. |
| Approach: | They propose a task to enable a single model to answer as many different domains of questions as possible . they break the task down into the integration of three key abilities . |
| Outcome: | The proposed model can answer as many domains of questions as possible, the authors argue . the proposed model generalizes well to three extra zero-shot datasets, and the results are published. |
Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) demonstrate strong visual question answering (VQA) capabilities but are shown to hallucinate. |
| Approach: | They propose three confidence-based methods to enhance LVLMs' perception . they propose probabilistic and consistency-based signals are more reliable indicators . |
| Outcome: | Experiments on three LVLMs across three VQA datasets show that LVLs possess a reasonable perception level but there is room for improvement. |
M3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-based VQA benchmarks focus on coarse-grained categories and simple reasoning over single entities. |
| Approach: | They propose a knowledge-based Visual Question Answering benchmark to enhance multimodality evaluation. |
| Outcome: | The proposed benchmark improves evaluation of multimodal large language models in fine-grained multimodal entity understanding and complex multihop reasoning. |